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I. INTRODUCTION 



The recent draft of a revision to Department of Defense 
(DOD) Directive 5000.1 suggests that the Pentagon is moving 
into a new era of program management. The groundwork for 
this revision to David Packard's earlier attempt to establish 
a firm policy base for the DOD procurement has been well 
laid by the reports of the Commission on Government Procure- 
ment and the Acquisition' Advisory Group, and by Office of 
Management and Budget Circular A-109. This modification of 
Directive 5000.1 places greater emphasis on the necessity 
to evaluate proposed new weapon systems on the basis of 
reconciled mission needs and requires an acquisition process 
in which alternative concepts/solutions are investigated in 
detail. The degree of program oversight by the Office of the 
Secretary of Defense (OSD) as proposed has been increased by 
the addition of another decision milestone, "DSARC 0", at 
which point the initiation of the conceptual effort will 
require Secretary of Defense approval. 

Faced with mounting public pressure for greater control 
of costs and performance following some notable examples of 
forecasting errors like the C-5A and LHA programs, DOD 
appears to have responded by striving to exorcise every 
possible bit of uncertainty from the conduct of weapon 
system acquisitions prior to the committment of major 
resources. The desired result would be the procurement of 
systems which satisfy basic operational requirements by a 
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firm date and at a reasonable and predetermined price. 

Although written several years prior to the proposed 

policy update, J. Ronald Fox’s words have an * I ’ve-seen-it- 

all-before' air to them as he comments in Arming America on 

the changing patterns of military acquisition management. 

Since 1969 there have been further attempts to 
improve other aspects of weapons acquisition 
management. These include the establishment of 
"milestones" for major programs (i.e., dates for 
the accomplishment of crucial program objectives) ; 
the synchronization of management personnel 
changes with program milestones; a reduction in 
the numbers of letters of contract; the improvement 
of training programs for program management 
officials; a formal review procedure for each new 
weapon system's operational specifications. The 
object of the review would be to insure that 
program requirements are reasonable and attainable, 
and that prototype systems are developed and 
tested before a production program is approved. 

Serious doubts about the successful implementation 
of this proposal have troubled Defense Department 
observers, however. . .Many procurement officials 
in the Defense Department are, after all, aware 
that the present system has faults. But despite 
a steady succession of studies and recommendations, 
the procurement process has remained impervious to 
structural reform. 1 

Certainly only' time will prove Fox right or wrong, but there 
is some evidence submitted by OSD that cost control has at 
least received greater attention among the major programs.^ 



1 Fox, J. R., Arming America , p. 456, Harvard University 
Press, 1974 

2 In a memorandum originating in the Office of the 
Director, Planning and Evaluation (OSD) dated 9 August 1976, 
it was reported that weapon system cost growth had declined 
generally in the last twelve months. 
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The purpose of this study was twofold. The first was 
to establish an historical perspective of major programs 
reaching fruition in the early to mid-1970's which will 
provide a means of comparison for future studies of program 
success, and thus perhaps support Fox's view. While the 
meaning of "program success" might well vary, it is here 
defined as the degree to which a program met initial 
estimates of cost, schedule and performance criteria. The 
first goal was met essentially by listing those programs 
studied with their associated degrees of success in 
Appendix C. 

The second desired result was to investigate the validity 
of certain hypotheses about program management. Specific- 
ally, the approach taken in this case was to highlight gross 
differences in program outcomes on the basis of shared 
program characteristics, such as program size, initial 
funding and technical advance. Associated with this aim was 
an extensive correlation analysis in which the predictability 
of program success was determined from program initial 
conditions when certain variables such as service sponsor, 
weapon system type and life-cycle phase were controlled. 

A. UNCERTAINTY IN WEAPONS SYSTEM DEVELOPMENT 

A. W. Marshall and W. H. Meckling of the Rand Corporation 
addressed the problem of the predictability of cost, 
schedule and performance estimates associated with development 
programs in a report prepared for the U. S. Air Force in 1959. 
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Their conclusions regarding the uncertainty of new develop- 
ments were: 

1. Early estimates of important parameters are usually 
quite inaccurate in two respects. First of all, they 
are biased toward overoptimism, and secondly, their 
error is subject to a large variation. 

2. The accuracy of estimates is a function of the 
stage of the system life-cycle at which time the 
appraisal was made. 

The authors suggested a relationship between developmental 
estimates and the military world of R&D decision making, 
but little if any analysis was provided in their report 
other than to say that a problem in the way estimates are 
generated does exist. Also, the number of programs 
reviewed appeared rather small (22) and they were all Air 
Force sponsored. 

Following the above report was the work of M. J. Peck 
and F. M. Scherer, The Weapons Acquisition Process: An 
Economic Analysis which was published in 1962. This bell- 
wether study of the business of arms procurement improved 
noticeably on Marshall and Meckling’s efforts to gain an 
understanding of the unpredictable nature of weapons 
acquisition. 

A major thesis of this study is that the weapons 
acquisition process is characterized by a unique 
set of uncertainties which differentiates it from 
other economic activity. To be sure, uncertainty 
is a pervasive feature of all economic activity, 
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and most of the uncertainties in weapons acquisition 
have their commercial counterparts. But there is a 
uniqueness in both the magnitude and diverse sources 
of uncertainty in weapons acquisition. 3 

The operational definition of uncertainty was provided by 

the authors as "the relative unpredicatability of the outcome 

of a contemplated action". 

Herein lies a problem which has confronted those 

researchers who have attempted to subject the success of 

program outcomes to statistical analysis based upon a 

comparison of initial estimates and demonstrated performance. 

Peck and Scherer summarized the difficulty in this way. 

Let us begin by attempting to ascertain the degree to 
which the outcomes of weapons programs are predictable, 

A rough measure is obtained by examining three crucial 
kinds of weapons program outcome predictions: (1) qua- 

lity, or the expected technical performance and the 
reliability of the resulting weapon system; (2) develop- 
ment time, or the interval between the start of a 
development effort and the availability of operational 
weapons; and (3) the cost of development. Characteris- 
tically, predictions of these variables are made by 
contractors and/or buying agencies early in a weapons 
program and are recorded in written documents. The 
predictability of the variables can be determined by 
comparing’ the early predictions with actual quality, 
time and cost outcomes. 

Nevertheless, comparisons of this sort are properly 
viewed with considerable reservation, for there are 
extreme difficulties in obtaining satisfactory data. 
Neither the original estimates nor the final outcomes 
are completely comparable between programs, nor are 
they even necessarily representative of the true state 
of affairs within a given program. 4 



3 Peck, M. J. and Scherer, F. M., The Weapons Acquisition 
Process: An Economic Analysis , p. 17, Research Division, 
Graduate School of Business Administration, Harvard University, 
1962 



4 Ibid., p. 19 
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The reasons given did overlap to an extent with those 
reported by Marshall and Meckling. 

1. Initial predictions were made at varying points 
in the maturity of different programs. 

2. Initial estimates were optimistically low and 
in some cases deliberately inaccurate. 

3. Performance outcomes could not be measured 
quantitatively . 

4. Time of availability outcomes were qualitatively 
assessed . 

5. Cost outcomes were blurred by arcane bookkeeping 
procedures, whereby developmental costs were buried 

in production expenditures or post-production support. 

In a criticism of the Selected Acquisition Report (SAR) , the 

Blue Ribbon Defense panel questioned the validity of before- 

and-after comparisons futher. 

The basic approach to the SAR is the establishment of 
a baseline of estimated costs, schedules and technical 
performance, and the subsequent measurement of the 
present status against this baseline. . .Successfully 
predicting the course of development of a new weapon 
system is uncertain at best. The long period of time 
involved introduces unpredictable changes as outside 
events and circumstances shift during the five to nine 
years it usually takes to acquire a new weapon. The 
development process itself contains hidden unknowns. 

The original estimates of cost, schedule and technical 
performance of a weapon system can be made with con- 
siderable skill and with total honesty, but they remain 
only estimates, the worth of which can be determined 
only by the future unfolding of events. The SAR's 
tend to treat the original estimates as accurate pre- 
dictions and to measure subsequent events in the dev- 
elopment against the standard of the original estimates. 
There are two serious consequences of this procedure. 
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...The overwhelming concentration now appears to be 
on maintenance of the costs and schedule within the 
original estimate. Concern with the quality of the 
weapon and its ability to perform an essential mission 
are not presented in the SAR. . .Inhibition against 
change is the second serious consequence of the present 
SAR system. 5 

The picture is not so bleak as one would be led to 
believe. Peck and Scherer pointed out the other side of 
the ledger as well in their analysis of the processes of 
weapons development. 

To examine the effectiveness of weapons program execution 
in an operational way requires a meaningful criterion of 
effectiveness. One possibility is to evaluate actual 
time, cost and quality results against original predic- 
tions. In particular, original time, cost and quality 
predictions do not necessarily reflect optimal choices. 
Still, the analysis of variances from original predic- 
tions has several values. For one, it is useful method 
of pinpointing the problems which affect program out- 
comes. Furthermore, to the extent that major national 
security decisions are made on the basis of original 
predictions, sizeable variances from those predictions 
may imply a serious misallocation of resources. 

A greater implication of the position that comparative studies 
are of no utility is that without comparisons, the effective- 
ness of reforming activities within the weapon system 
acquisition process must be left entirely to intuition. 

Peck and Scherer did attempt to pinpoint problems 
affecting program outcomes by correlating Marshall and Meck- 
ling's criteria of program success with a largely qualitative 



5 Blue Ribbon Defense Panel, Report to the President and 
the Secretary of Defense on the Department of Defense , Vol. 1, 
p. 125, U.S. Government Printing Office, 1970 

6 Peck, M. J. and Sherer, F. M. , The Weapons Acquisition 
Process: An Economic Analysis , p. 428, Research Division, 
Graduate School of Business Administration, Harvard University, 
1962 
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collection of explanatory variables. The results were 
characterized by a lack of statistical significance which 
might be explained by a reliance on parametric procedures. 
However, the results did indicate that definite relation- 
ships existed between program characteristics externally 
imposed on the project, such as urgency, the importance of 
minimizing costs, state-of-the-art exploitation and program 
results . ^ 

This study was conceived in part as a redirected con- 
tinuation of the above aspect of Peck and Scherer's work; 
redirected in the sense that the choice of independent 
variables was limited to quantitative measures of a program 
(e.g. , unit cost, total program cost, development expendi- 
tures and program length) which were then applied to a 
larger sample of programs using nonparametric analyses. 

However, more importantly this thesis has attempted to 
ascertain the validity of certain general beliefs of the 
nature of program management and its effect on program out- 
comes. Four separate hypotheses have been presented, each 
in its own right questioning the predictability of project 
success when considered as the result of a single determinant 
variable. 8 in short, the thesis has erected a model of the 
acquisition process in which the management skills and 

7 Peck, M. J. and Sherer , F. M,, The Weapons Acquisition 
Process: An Economic Analysis , ch. , Research Division, 
Graduate School of Business Administration, Harvard University, 
1962 

8 A fifth hypothesis was posed to test the data source 
for bias. 
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practices of a system's program office staff or individual 
manager have been ignored as predictors of program results. 

The model was further refined by suppressing the externally 
imposed noise of program contractions (expansions) and 
economic escalation. The hypothesis testing served to 
illustrate one dimensional differences in acquisition programs 
(i.e., cost growth differences in programs differentiated 
by project cost alone) . 

In addition, a series of correlation tests were conducted 
in which the fundamental differences in program outcomes were 
further tested by controlling additional program character- 
istics such as service sponsor or weapon system type . The 
purpose of the correlation was twofold, first to establish 
whether the controlling process accented the gross differences 
identified by .the hypothesis testing, and secondly, to identify 
among the program facets of size, length and risk, effective 
predictors of program outcomes. 
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II. METHODOLOGY 



A. SELECTED ACQUISITION REPORT 

Recognizing the problem of how program maturity may 
distort the accuracy of early forecasts, it was considered 
essential that some instrument adjust initial estimates for 
all programs to a common beginning, and also to allow for the 
adjustment of these forecasts in the light of a lengthy 
program life. Such an instrument was the Selected Acquisition 
Report (SAR) . 

The SAR serves as a comprehensive, summary acquisition 
status report for the "highest levels of management within 
the Department of Defense and (is) the basis for reports to 
the Congress and other government agencies including the 
Government Accounting Office (GAO)".^ Compiled at the close 
of each quarter, the SAR tracks the success of each major 
weapons system acquisition in the areas of cost, technical 
performance and schedule. 

In each of these areas, the report offers a comparison 
between current estimates for program success and the develop- 
ment, or initial, estimates. Sandwiched between these two 
expectations is yet a third, the approved program estimate 
which in practive varies only slightly, if at all, from the 
development estimate. The development estimate is established 

9 Secretary of the Navy Instruction 7000.3 
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once the Secretary of Defense grants his approval for the 
program to move into full scale development, and generally 
has as its basis the Decision Coordinating Paper (DCP) . 

The development estimate for procurement may be reviewed once 
the production contract has been let, but otherwise, the 
estimate may not be changed without permission of the Assis- 
tant Secretary of Defense (Comptroller) . Fox points out 
the necessity of this last requirement as he quotes a DOD 
review of management control practices. 

This problem has been common enough to acquire the 
esoteric but descriptive label, "the rubber baseline". 
Translated, this means that the contractor managers 
would periodically make retroactive adjustments to 
cost estimates and schedules so that plans would 
equal actuals. . .Obscuring overruns in this fashion 
contributed to our being caught short on funding for 
two major programs last fiscal year. 10 

The approved program estimate is also based upon the DCP, or 

in its absence, a Program Memorandum or Program Change 

Decision. As these formal statements of changes in program 

goals are forthcoming, the estimate will necessarily be 

revised. The current estimate represents the program 

manager's most realistic assessment of program status. As 

the program matures, this appraisal will more and more equate 

to demonstrated performance or the actual completion of 

schedule milestones. The 31 March 1976 SAR for the A-7E 

project is provided as an example in Appendix A. 

The 30 June 1976 SAR was the principal, primary reference 



10 Fox, J. R. , Arming America , p. 413, Harvard University 
Press, 1974 
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source of project data for this thesis. In fact, because of 
the completeness of the report and the clarity of its format, 
whether or not an acquisition program was included under the 
reporting system determined whether that program would be 
included in the analysis. Thus, this determination excluded 
a number of weapon systems presently in either development or 
procurement, principally in the areas of electronic counter- 
measures and naval gunnery. 

B. ANALYTIC METHODS 

1 . Hypothesis Testing 

1. Large programs have a tendency to incur greater 
cost and schedule growth in absolute and relative terms than 
do smaller programs . 

One of the initial, underlying assumptions of this thesis was 
that not only -will weapon system acquisitions experience cost 
growth, but that for system with large program costs, the 
growth will be correspondingly greater than for those less 
expensive projects. Reasons for the explanation of the cost 
growth phenomenon have been postulated again and again by 
critics of military procurement (e.g,, Peck and Sherer, Fox). 
Not the least frequently observed reason is that of over- 
optimism. It would seem reasonable that large, expensive 
programs would require a certain amount of "selling" on the 
part of sponsors to ease the difficulties of higher level 
DOD and Congressional approval. This "selling" often took 
the form of understating the eventual program cost, and with 
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it the technological risk involved in the development, not 
to mention the date on which the system enter service. 

Whether overoptimism was the result of deliberate policies 
on the part of the sponsoring agency is something of a moot 
point. Peck and Scherer, however, did point out that the 
practice has been accepted as a matter of fact within the 
defense industry. 

Quite commonly, this strategy includes submitting 
proposals which in one or more aspects are highly 
optimistic. Indeed, the propensity to 'but into' 
attractive new programs with optimistic quality, 
time, and/or cost estimates is perhaps as much as 
an industry practice in advanced weapons acquisition 
as list price cutting is in automobile retailing, or 
as the advertising of loss leaders is in department 
store operations . i 1 

Concurrent with the overoptimism that acompanies large 
programs, it was presumed that there would be difficulties 
encountered in the area of cost control simple because of 
program size, an extension of the belief that with size one 
inherits waste and a certain amount of uncontrollability. 
Unfortunately, it is impossible to distinguish explicitly 
between optimism or waste based on data gathered from the 
SAR' s. 

2 . Programs with a larger development phase budget 
have a greater tendency to achieve initial cost, 
performance and schedule estimates . 

An acedemic approach to the study of project management has 
stressed the importance of "front-end loading" the acquisition 



Peck, M. J. and Scherer, F. M. , The Weapons Acquisition 
Process: An Economic Analysis , p. 413, Research Division, 
Graduate School of Business Administration, Harvard University, 



1962 
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effort, citing the lever effect which early commitment of funds 
has upon program performance.^ it would appear reasonable 
that programs which enjoy an extensive development period 
would suffer the least uncertainty in the expected level of 
program outcome, since a greater percentage of the "unknowns" 
which plague any program would be uncovered. 

3 . Programs with greater technical risk tend to incur 
greater cost and schedule growth than do less risky 
programs . 

This hypothesis is associated with the second contention in 
that programs which represent a great deal of technical risk, 
which require a more complex and sophisticated technical 
effort (as perceived from the outset of the program) , would 
be more susceptible to failure. 

4 . Multiple-source contracted programs enjoy an advan- 
tage in program success over sole-source contracted 
programs . 

For the purpose of this analysis, those programs which had 
more than one" prime contractor regardless of whether the 
systems were in procurement or development were considered 
multiple source programs. In the case of air-craft systems, 
if there were competing contractors for either the airframe 
or engine designs, the program was considered to have a 
"second source". Frederick Sherer in his study of the 
economic incentive in the weapons system acquisition process 

12 ihe effect was originally suggested to this student 
by Professor Melvin Kline, U. S. Naval Postgraduate School. 
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documented the significant cost reductions which developed 
during the joint production of military aircraft in World 
War II. Although not in as much detail as Scherer, Fox 
touched upon the benefits of "second sourcing”, reporting 
that "it would be reasonable to assume a 25% reduction in 
contract price can be achieved in many, if not most, cases 
when competition is introduced. "13 it would appear that 
this is one area in which weapons procurement responds 
to traditional market forces. The fourth hypothesis seeks 
to determine first, whether the evident cost successes do 
exist in a significant number of cases, and second, whether 
the advantage of multiple-source contracting is extended 
to the areas of performance and schedule. 

5 . The prospect of program success dimenishes with 

increasing maturity among programs . 

This last hypothesis was posed in order to test the validity 
of the SAR in overcoming the criticisms alluded to earlier 
about the non'comparability of programs in differing stages 
of their life-cycle. For example, early in the life of a 
program, before the realities of the management or technical 
task at hand are more apparent, one would expect that current 
estimates of project outcomes would agree with initial 
assessments. But, as the program achieves greater maturity, 
and uncertainties arise, the gap between the two expectations 

13 Fox, J. R. , Arming America , p. 256, Harvard University 
Press, 1974 
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would be sure to widen. 



Each hypothesis was tested using a one-tailed Mann- 
Whitney U Test (a nonparametric one way analyses of variance) 
in which the means of program outcomes were compared. For 
the most part, the sample of 48 programs would be divided 
along the median of the program variable in question, except 
in the case of the fifth hypothesis where whether a program 
was in development or procurement became the classifying 
criterion. The results of these tests are presented in 
Chapter III. 

2 . Correlation 

As stated earlier, associated with the hypothesis testing 
was a desire to examine the correlation between the output 
variables of cost, performance and schedule, and initial 
program estimates of program length, cost and technical risk 
when specific program characteristics were controlled for 
explicitly. The thesis employed a nonparametric correlation 
proceudre offered as a subprogram in SPSS: Statistical 
Package for the Social Sciences by Norman H. Nie, et al. 

The following characteristics were considered and controlled 
for in deriving correlations between program outcomes and 
initial estimates: 

1. All forty-eight programs together without any 
differentiation; 

2. Programs differentiated by the maturity of their 
design (i.e., whether the program was in development or 
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had advanced to procurement) ; 

4. Programs differentiated by system type, ship, 
aircraft, missile or miscellaneous (tracked vehicles, 
fire-control systems, etc.); 

5. Programs differentiated by service sponsor, Navy, 
Army, or Air Force. 

The first category of 'undifferentiated programs' was 
included as a control feature to serve as a means of com- 
parison for the following four program characteristics. 

Items two and three were included as part of follow-up tests 
to the verification of the fourth and fifth hypotheses. 

The last two program characteristics, system type and service 
sponsor, were included because they are frequently used as 
means of ordering weapons systems. Also, there are sub- 
stantial inter-service differences in system acquisition, 
and as a result, there are natural, virtually instinctive 
inclinations to make comparisons between the programs of 
each service. 

Additionally, the intercorrelations among program out- 
comes were examined to see if any light could be shed on the 
trade-offs between cost, performance and schedule. The 
correlation results are presented in Chapter III. 

C. SELECTION OF INDEPENDENT VARIABLES 
1 . Program Cost 

For the purpose of the analysis, it was considered 
essential that each of the forty-eight programs be considered 
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on an equal footing, that is, considered from the same base 
year. In this way, comparability between programs on the 
basis of total program cost would be maintained. The base 
year selected was 1974 since all but two programs (added late 
in the thesis) predated that year. Each of the programs 
reported development estimate program costs in their own 
base year dollars, which in turn were converted to 1974 
dollars. The method of accomplishing this conversion was 
not very satisfactory, but had to suffice for lack of a 
superior alternative. From a Direct Budget Plan (TOA) 

History for the Department of the Navy, an implicit price 
deflator for the fiscal year 1964 through 1977 was derived. 
The unsatisfactory aspects of this source are evident when 
one considers that the TOA history included nonprocurement 
appropriations' such as OM&N as well as the procurement 
related appropriations, RDT&E, SCN, etc. The U. S. Bureau 
of Economic Analysis is in the process of determining 
deflators for each of the basic defense industries, but 
results will not be forthcoming until late 1977. 

A copy of the TOA history and an example of the manner 
in which the implicit price deflators for each of the 
program base years was derived is provided in Appendix B. 

2 . Program Unit Cost 

Unit cost was selected as an independent variable 
on the assumption that program success (primarily in the 
area of cost) could not be adequately explained by program 
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size *alone. For example, a program line the A-7E in which 
the Navy is procurring a proven aircraft design with up- 
dated avionics, and the F-14A program which represents a 
completely new weapon system, could have conceivably identical 
program costs provided sufficient quantities of the A-7E 
were purchased. However, the true nature of both programs 
would be obscured. Of the two, clearly the F-14A with a 
program unit cost of $16. 8M (1974$) is the more ambitious 
undertaking, and the lesser unit cost of the A-7E, $4.1M 
in 1974 dollars, would reflect this. As in the case of 
program cost, development estimates of unit costs were 
translated into 1974 dollars. 

3 . Development Investment 

The development investment for each program was 
derived as the „ ratio of the level of development spending 
to total program as foreseen at DSARC II. No judgement of 
the manner of (or purpose for) development spending was 
attempted. The variable was intended to serve as a com- 
plement to program unit cost, that is, a measure of the 
technological advance of the project. Some difficulties 
were encountered in those reports for nuclear powered 
men-of-war where propulsion system development costs 
attributed to both the Navy and the Energy Research Devel- 
opment Agency (ERDA) were not included in the cost section 
of the SAR's, but rather in a SAR addendum. Consequently, 
at first glance a project as large and complex as NIMITZ 
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(CVAN 68) appears to be an off-the-shelf product, despite 
the fact that her two reactor propulsion plant represents 
a first for the surface Navy and is hardly a trivial tech- 
nological step. Adjustments were made for cases such as 
this by adding the additional estimated development cost 
to both the level of development funding and the tot^.1 
program cost. 

4 . Program Length 

Of all the explanatory variables, time was the most 

subjectively determined. Initial program length, measured 

ostensibly from the initiation of the project to an initial 

operational capability date (IOC) proved to be a bit 

troublesome to extract from the SAR’s. Earlier researchers 

have encountered similar problems. 

Time of availability predictions are often made in 
terms of 'an ’initial operational capability’ - a 
decidedly qualitative concept. Does one missile 
on a lonely launch pad constitute an ’initial opera- 
tional capability’? Or do 30 missiles when their 
reliability in time of need is doubtful?!^ 

Fortunately, the term IOC is a common one within DOD and in 

a large number of cases, those dates were reported explicitly. 

In the case of Navy shipbuilding programs, a simplifying 

rule was established whereby the date on which the lead ship 

of the entire class in question became available for 



14 Peck, M. J. and Scherer, F. M., The Weapons Acquisition 
Process: An Economic Analysis , p. 20, Research Division, 
Graduate School of Business Administration, Harvard University, 



1962 
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operational deployment was chosen arbitrarily as the "IOC” 
date. There are obvious draw-backs to this method when the 
entire class is considered as any student of ship construc- 
tion can point out. Namely, the greatest schedule slippages 
often occur during follow-ship production. Still, the 
availability of the lead ship does represent the realization 
of a design, and in order to maintain some parity with other 
systems produced more rapidly, the consistency of the above 
method was considered essential. 

The other end-point of program length, the initiation 
of the project, was even more difficult to determine. In 
most cases, a date corresponding to the advent of the system 
life-cycle was selected as the starting point. More often 
than not, this date was the date of approval for the system 
requirements/characteristics. There was a distinct lack of 
uniformity noted among the various programs in the reporting 
of this schedule milestone. In a few cases programs were 
considered underway with an all inclusive 'commenced 
engineering development' or 'initiated validation phase'. 

D. SELECTION OF DEPENDENT VARIABLES 
1 . Cost Growth Factor 

Fortunately, a recent revision of the SAR system 
directed that development and current cost estimates be 
reported in constant year dollars as well as in escalated 
figures. This change has made the pinpointing of true cost 
growth that much easier. It came as something of a shock 
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to discover that little or no provision for escalation was 
included in the development estimate of many programs. 

Where some accounting for escalation was made, invariably 
the Inflation rate was unrealistically low. 

The office of the Assistant Secretary of Defense 
(Comptroller) publishes a "SAR Program Acquisition Cost 
Summary" (see Appendix C) once each quarter's collection of 
individual reports is complete. From this summary it is 
possible to determine each program's cost growth both in 
dollars and as a percentage of original estimates. In each 
instance, the resultant figures are adjusted by OSD analysts 
to negate the effects of inflation and program quantity 
changes. Unfortunately, the relative measure of program 
cost growth is related def initionally to the original 
program cost estimate as shown below. 

Cost growth (%) = Cost Changes 

Development Estimate + Quantity Changes 

(All entries are in base 
year dollars) 

One would expect a negative correlation between cost growth 
(%) and program size, expressed above as the ’development 
estimate’ . The extent of the correlation would depend on the 
degree to which the Cost and Quantity Changes varied with 
total program cost. To aid in clarifying the effect of 
program size on cost success, it would be helpful to have the 
assistance of an absolute measure of growth, namely Cost 
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Growth (?) which is the dollar value of the overrun. This 
measure is determined as shown below. 

Cost Growth ($) = Cost Growth (%) -1 x Development 

Estimate 

Hence, a negative "growth" would indicate an underrun. 

2 . Performance Achievement Factor 

Navy and Air Force concepts of system effectiveness 
were presented in the 1960 's as a measure of how well a 
system performed its intended function. This figure of 
merit was derived originally as a companior to cost effective- 
ness in the hope that together the two would serve to deter- 
mine whether a system undergoing design would satisfy most 
efficiently those operational requirements which spawned it 
in the first place. Despite interservice differences- in the 
wording of what was meant by system effectiveness, the 
measure (expressed as a probability) was generally conceeded 
to have three basic components; 

1. the performance capability of the system; 

2 . the operational readiness or availability 
that is, its ability to start performance of a 
mission when called upon to do so; 

3. the continued capability of the system to 
perform, or its mission reliability or dependa - 
bility . 

In time however, it became apparent that models of system 
effectiveness were not as useful as originally conceived. 
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The computations were messy and highly subjective. For 
example, how much does the failure of a single component 
degrade the effectiveness of a system? Or even more basic, 
what constitutes a failure? Since the derived probabilities 
were by no means "hard numbers"', they were mistrusted by 
managers, and the computational aspects of system effective- 
ness fell into disuse. 

Given that an evaluation of system performance is essential 
to any analysis of program success, and in the absense of a 
readily available performance factor counterpart to cost 
growth, a simplistic method of marking performance using the 
three basic elements of system effectiveness where possible 
was devised. From SAR data on the technical/operational 
characteristics of the system in question, it was possible 
to contrast current estimates of system performance with 
development estimates. Generally speaking, the performance 
achievement factor was determined by assessing a weighted 
average of major performance areas such as speed, time-on- 
station, maintainability, system accuracy, etc. A value of 
performance less than 1.0 indicates that the system achieved 
better results than originally estimated. An example of how 
this computation was made is provided in Appendix C. 

3 . Schedule Overrun Factor 

As in the case of attaining performance goals, the 
schedule overrun factor was determined by the ratio of 
current to development estimates of program length. However, 
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like cost growth (%) , shcedule slippage (%) is related 
inversely to program length. 

Schedule slippage (%) = Actual (Current Estimate Of) 

Program Length 

Development Estimate of 
Program Length 

Hence, a negative correlation would be expected. To minimize 
this dependency, a fifth output variable, schedule slippage 
(months) was introduced. In a similar fashion as cost 
growth (?) , this variable measures the actual slippage of the 
IOC data in months. 
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III. RESULTS 



A. HISTORICAL PERSPECTIVE 

Appendix C has listed those programs from which program 
data was obtained for the analysis. With the exception of 
POSEIDON, all the programs are currently included under the 
SAR system. (POSEIDON reports were terminated in June 1975 
following completion of system deployment and the subsequent 
disestablishment of the program office) . For the convenience 
of the reader, the programs are ordered by service sponsor. 

An examination of the outcome means for each service indicates 
that perhaps Navy programs enjoyed a greater achieved per- 
formance at greater cost and shcedule growth than did Army 
and Air Force projects. A sample comparison between outcome 
means revealed that the NAVY - USAF differences in all three 
categories were significant, while the pertinent ARMY - NAVY 
disparity was limited to the area of cost alone. 

The results of the NAVY - USAF comparison are not fully 
understood. Sample comparisons of Navy programs (less ship 
systems) and the Air Force projects shown did result in less 
significant cost differences. Additionally, when missile 
systems (which historically have proven to be technically 
more successful than other system types) were excluded from 
both samples, the performance difference between services 
was still evident. No explanation for this is readily 
apparent, nor was there any system type peculiar to a 
single service that could account for the Air Force advantage 
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in meeting deployment schedules 
B. HYPOTHESIS TESTING 

1 . Large programs have a tendency to incur greater cost 
and schedule growth than do smaller programs . 

Table 3-1 displays the difference in the means of 
program outcomes between large and small projects. The column 
marked 'significance' indicates the degree to which the 
differences were statistically significant. In instances of 
no significance, the column entry is NS. Subsequent tables 
in this section are arranged similarly, 

TABLE 3-1 

COMPARISON OF PROGRAMS BASED ON PROGRAM SIZE 



LARGER PROGRAMS 




SIGNIFICANCE 


SMALLER PROGRAMS 


Cost Growth (%) 


1.13 


0.003 


Cost Growth (%) 


1.29 


Performance 


0.99 


NS 


Performance 


1,00 


Schedule Slip 
(%) 


1.16 


0.005 


Schedule Slip 
(%) 


1.44 


Schedule Slip 
(Mos . ) 


12.6 


0.005 


Schedule Slip 
(Mos . ) 


25.2 


Cost Growth ($) 534 . OM 


0.006 


Cost Growth ($) 163 .7M 


The results appear to 


provide mixed 


support for the first 



hypothesis. Ignoring the differences in relative cost growth 
for reasons given in Chapter II, note the differences in 
schedule overrun. Historically, larger programs have done 
significantly better in meeting original schedules, both in 
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a relative and absolute sense. The lack of significance 
observed in the area of achieved performance is a result 
that will be repeated elsewhere in this chapter. Overall, 
the nonparametric analyses of variance were ineffectual in 
accounting for the variance in system performance. 

In the area of cost growth ($), larger programs 'did not 
appear to hold the line against overruns. The reasons for 
this may have included a tendency for the managers of large 
(and therefore important) programs to throw money at 
problems, particularly schedule problems, until a solution 
is found. Once a major program has been funded, rarely 
has it had to do without additional resources if required. 

2. Programs with more development funding have a greater 
tendency to achieve initial cost, performance and 
schedule estimates 

TABLE 3-2 

COMPARISON OF PROGRAMS ON BASIS OF LEVEL 
OF DEVELOPMENT FUNDING 



HIGHER PROGRAMS 


SIGNIFICANCE 


LOWER PROGRAMS 






Cost Growth (%) 


1.15 


0.040 


Cost Growth (%) 


1 . 


27 


Performance 


0.99 


NS 


Performance 


1 . 


0Q 


Schedule Slip 
(%) 


1.24 


0.002 


Schedule Slip 
(%) 


1 . 


42 


Schedule Slip 
(Mos. ) 


15.94 


NS 


Schedule Slip 
(Mos. ) 


21 


.7 


Cost Growth 


459. 9M 


NS 


Cost Growth 


234 


.7 



($) ($) 
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Perhaps the most important comparison in Table 3 - 2 is that 
which contrasts relative cost growth. During the analysis 
of program data, a very high (0.750-1.000) positive corre- 
lation was found to exist between program cost and develop- 
ment funding in a significant number of cases. It was likely 
that the two were measuring essentially the same character- 
istic, program size. Thus, one would expect Tables 3-1 and 
3 - 2 to bear a resemblance to one another. In terms of 
relative cost growth and schedule slippage, this similarity 
exists. What is crucial about the results of this second 
sample comparison is that the level of development funding 
has a significant negative impact on relative cost growth. 
Bearing in mind that the development cost is also an indirect 
measure of program size, this is the first instance of an 
indication, not confounded by definitions of the variables, 
that suggests program size may inhibit the degree of cost 
growth. Further signs of the efficacy of early and sub- 
stantial investments are the differences in relative schedule 
slippage and the absence of any significance to the difference 
between the absolute cost growth for each sample. 

3 . Programs with perceived technical risks tend to incur 
greater cost and schedule growth than do less risky 
programs 
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TABLE 3 



3 



COMPARISON OF PROGRAMS ON 
BASIS OF DEVELOPMENT INVESTMENT 



HIGHER PROGRAMS 




SIGNIFICANCE 


LOWER PROGRAMS 




Cost Growth (%) 


1.18 


NS 


Cost Growth (%) 


1.24 


Performance 


0.97 


NS 


Performance 


0.97 


Schedule Slip 
(%) 


1.32 


NS 


Schedule Slip 
(%) 


1.33 


Schedule Slip 
(Mos . ) 


19.4 


NS 


Schedule Slip 
(Mos. ) 


18.2 


Cost Growth ($) 


349. 8M 


NS 


Cost Growth ($) 


340.3M 



The weakness of development investment as a explanatory 
variable is apparent in Table 3-3. No significance can 
be attributed to the difference in outcomes between programs 
with a higher 'development/committment and those with less. 

A second comparison between the extremes of each group, the 
highest and lowest quartiles, was made with similar null 
results . 

4. Multiple-source contracted programs enjoy an advan- 
tage in program success over sole-source contracted 
programs 
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TABLE 3 



4 



COMPARISON OF PROGRAMS ON BASIS OF 
PRODUCTION/DEVELOPMENT SOURCES 



SOLE-SOURCE 




SIGNIFICANCE 


MULTIPLE SOURCE 


Cost Growth (%) 


1.21 


0.040 


Cost Growth (%) 


1.14 


Performance 


0.99 


NS 


Performance 


1.02 


Schedule Slip 


1.30 


NS 


Schedule Slip 


1.44 


(%) 






(%) 




Schedule Slip 


17.8 


NS 


Schedule Slip 


24.7 


(Mos. ) 






(Mos . ) 




Cost Growth ($) 


382. 3M 


0.049 


■ Cost Growth ($) 


103. 4M 


The findings of 


Peck 


and Scherer were sustained by the 


Mann- 


Whitney test. 


In terms of relative 


and absolute cost 


growth, 



second-source programs did better than their sole-source 
counterparts. ' Although a small class of projects (7) , the 
multiple source programs included Army and Navy projects of 
all system types. In regard to the achieved performance of 
both program types, this study found no significance in the 
difference of program outcomes. This is at variance with the 
findings of Rubin and Marquis' who determined that "sole source 
contractors do significantly better technically than contrac- 
tors who were selected by some form of competitive procedure . 
Rubin and Marquis' study is not precisely comparable to the 



Sloan, Alfred P., School of Management, Massachussetts 
Institute of Technology, Report #192-66, Critical Decisions in 
the Initiation of Development Projects , Rubin,, I. M. and 
Marquis', D. G. , June 1966 
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one undertaken here in that only development projects in 
which the technical performance was evaluated subjectively 
by management and engineering personnel, were considered. 
However, the contention that greater technical performance 
can be achieved in the absence of competition was not 
supported . 

5 . The prospect of program success diminishes with 
increasing maturity among programs 

TABLE 3 - 5(A) 

COMPARISON OF PROGRAMS BASED 
ON LIFE-CYCLE DIFFERENCES 



PROGRAMS IN PROCUREMENT 


SIGNIFICANCE 


PROGRAMS IN DEVELOPMENT 


Cost Growth (%) 


1.22 


NS 


Cost Growth (%) 


1.18 


Performance 


0.98 


NS 


Performance 


1.00 


Schedule Slip 
(%) 


1.33 


NS 


Schedule Slip 
(%) 


1.29 


Schedule Slip 
(Mos . ) 


18.7 


NS 


Schedule Slip 
(Mos. ) 


17.5 


Cost Growth ($) 


460. 6M 


0.012 


Cost Growth ($) 


213. 9M 



Table 3-5 (a) reveals that on a gross level the SAR is able to 
minimize the differences among programs that arise because 
of life cycle disparities. With the exception of program 
growth in dollars, programs in procurement and those still in 
development exhibited similar success. The exception of 
program growth is indicative that either uncertainties in 
schedule or producibility do still exist, and that the 
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development phase has not been completely successful in 
identifying and costing these aspects of the system. It is 
felt that the lack of significance between the two categor- 
ies can be attributed to allowing the SAR development 
estimate to be reviewed and updated when required once a 
production contact has been let. 

In the event that differences within groups were obscured 
by combination, the development and procurement classes were 
again divided, in the development cast by the age of the 
program, and in the case of procurement by whether the system 
had reached IOC. These two groups were then compared and 
the results presented in Table 3-5 (b). 

TABLE 3 - 5(B) 

COMPARISON OF PROGRAMS BASED ON 
LIFE-CYCLE DIFFERENCES 

PROGRAMS AT/PAST IOC SIGNIFICANCE RECENTLY INITIATED PROGRAMS 



Cost Growth (%) • 


1.22 


NS 


Cost Growth (%) 


1 . 


16 


Performance 


0.96 


0.025 


Performance 


0 . 


99 


Schedule Slip 
(%) 


1.29 


NS 


Schedule Slip 
(%) 


1 . 


35 


Schedule Slip 
(Mos. ) 


16.4 


NS 


Schedule Slip 
(Mos. ) 


24 


.1 


Cost Growth ($) 


523. 8M 


0.050 


Cost Growth ($) 


266 


. 5M 



The results were generally supportive of those shown in 
Table 3-5 (a) . Once again the difference in cost growth 
is significant; however, the difference in the means of 
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achieved performance was unexpected. One possible explana- 
tion is that the realities of a fully operational deployment 
might test system capabilities better than a formal evalua- 
tion period, principally in the important areas of relia- 
bility and maintainability. On balance, the findings in 
Tables 3 - 5(a) and 3 - 5(b) supported the contention that 
the SAR in its present form does overcome sufficiently the 
difficulties in comparing programs at various stages of 
completeness . 

C. CORRELATION 

The results of the nonpar ame trie correlation are presented 
in Tables 3-6 through 3 - 10(c). Overall, the results are 
not impressive. To be sure, there were individual cases in 
which the controlled characteristic produced effective 
results. For example, missile programs displayed an across- 
the-board effectiveness in predicting success. (See Table 
3-7 (b) ) . Why this is true of missile systems and not ships 
is not readily apparent. For whatever reason, there is a 
greater homogeneity among missile programs than among other 
groupings. Perhaps then, one could suggest that greater 
faith could be placed in the initial estimates of missile 
program success to the extent that they are valid indicators 
of future results. Characteristic, however, of the analysis 
was the finding of less than 50% of the correlations as 
significant. And, when those variable pairs with definitional 
relationships are discounted, the results are even more 
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nebulous. Additionally, the whole purpose of ordering 
systems by some specialized criterion variable to isolate 
the effects of program determinants is called into question 
when one recognizes that the category of "all programs" 
exhibited a greater number of significant correlations than 
a majority of the other instances when a program character- 
istic, like service sponsor, was controlled. Actually, the 
taxonomic analysis was weakened further by disparities in 
sample size among some of the classes of programs. The 
greatest difference was noted between the sole-source 
contracted programs (40) and multiple-source contracted 
programs (7) . It is not unexpected that those categories 
had by comparison more significant results than their smaller 
counterparts, since greater sample size lowers the level at 
which a relationship between variables can be deemed signifi- 
cant. 
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TABLE 3 



6 * 



ALL PROGRAMS 



Performance Cost 
Factor Growth 
Factor 
(%) 



Program 


Unit 


0.1193 


-0.2611 


Cost 




(0.038) 


Program 


0.1321 


-0.3488 


Cost 




(0.008) 


Program 


Development 

Investment 


0.0545 


-0.1415 


Program 


0.2822 


-0.2763 


Length 


(0.027) 


(0.032) 



Cost 


Schedule 


Schedule 


Growth 


Slip 


Slip 


Factor 


Factor 


Factor 


($) 


(%) 


(Mos . ) 

• 


0.1430 


-0.1668 


-0.0682 


0.4549 


-0.5408 


-0.3892 


(0.001) 


(0.001) 


(0.003) 


-0.0964 -0.0746 


-0.0608 


0.0999 -0.3898 
(0.004) 


-0.0664 



* Entries in parentheses indicate 
level to which Spearman correla- 
tion is significant. All others 
are not significant. 



Sample Size 48 



Program Outcomes (Mean) 



Performance 


0.992 


Cost Growth (%) 


1.210 


Schedule Slip (%) 


1.326 
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J-.tP „{.: s,".:. 





Performance 

achievement 


Cost 

rowth ( „i 


Most 

Growth Ct>) 


•Schedule 
Slip (7.) 


Schedule 
Slip (i'os 


Unit Cost 


HI A 




-0 .3766 


-0 . boo 7 
(0.05b) 


0.0678 


0.5439 

(0.065) 


Total Program 

"Obt 


M/A 




-0.7488 

(0.011) 


-0.4333 


-0.O27 
(0.035 ) 


-0.494 

(0.088) 


Deve lopment 
Investment 


N/A 




0.0890 


-0.2447 


0.4120 


0.5297 

(0.071) 


Program Lenuth 


N/A 




-0.3109 


- 0 . 4oU3 


-0.1362 


0.3613 








Mean 


Outcomes 






Sample Cize 
9 




Perf orma nee 
L.0U0 


Cost Growth 
1.220 


Scnedule Slip 
1.308 







T.J3UJ 3-7(13 


> 








i 


sssil;-: SYLTE; U 










Perrormance 


Cost 


Cost 


Schedule 


Schedule 




Achievement 


Grov.-th [%) 


Growth ($) 


Slip ('>) 


Slip (Mos. 


Jmt Cost 


-0. '612 


-0.3)05 


0.2953 


-0.4653 


-0.3940 






(U.055) 




(0.022) 


(0.048) 


Total Program 


-0.0123 


-0.4G77 


0.6017 


-0.6839 


1 

c 

• 

On 

Tj 

O 


Cost 




(0.025) 


(0.004) 


(0.001) 


(0.006 ) 


Development 


0.0432 


-0.5716 


-0.1861 


-0.3478 


-0.4118 


Investment 




(0.007) 




(0.072) 


(0.040) 


Program Lenqth 


0.4-71 


-0.2992 


0 ,43b 3 


-0.7703 


-0.4/80 




'.'.025) 




(0.040) 


(C.0C1) 


(0.022) 






f.ean Outcomes 






..anole izp 


Performance 


Cost Orowth 




•ch^dule Sun 



19 0.987 1.22o 1.509 



16 

No correlations with achieved performance were possible because each 
each of the programs in the sample was assigned a factor of 1.0 . 
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TABLE 3 - 7(C) 



AIBCRArT SYSTEMS 





Performance 

Achievement 


Cost 

Growth (%) 


Cost 

Growth ($) 


Schedule 
Slip (%) 


Schedule 
Slip (Mos.) 


Unit Cost 


-0.922 


0.1694 


0.3978 

(0.078) 


0.1046 


0.0871 


Total Program 
Cost 


0.3709 


-0.1606 


0.5341 

(0.025) 


-0.1246 


-0.0380 


Development 

Investment 


0.1554 


0.2381 


0.0551 


0.5062 

(0.032) 


0.4285 

(0.063) 


Program Length 


0.2001 


-0.2090 


-0.3055 


-0.1357 


-0.0112 






Mean Outcomes 






Sample Size 
14 


Performance 

0.991 


Cost Growth 
1.149 




Schedule Slip 
1.145 




- 


TABLE 3 - 


7(D) 








MISCELLANEOUS 


SYSTEMS 








Performance 

Achievement 


Cost 

Growth (%) 


Cost 

Growth (S) 


Schedule 
Slip (%) 


Schedule 
Slip (Mos . ) 


Unit Cost 


0.0 


U .2000 


0.5429 


-0.5000 


-0.1000 


Total Program 
Cost 


-0.1160 


-0.0286 


1.0000 

(0.001) 


-0.8000 

(0.052) 


-0.6000 


Development 

Investment 


0.2899 


-0.0857 


0.2571 


-0.3000 


0.1000 


Program Length 


-0.2500 


0.2029 


0.2899 


-0.6667 


-0.3591 



Mean Outcomes 



Sample Size 
6 



Performance 

0.998 



Cost Growth 
1.170 



Schedule Slip 
1.170 



44 



TABLE 3 - 3(A1 



Unit Cost 



Total Program 
Cost 



Development 

Investment 



Program Length 



Sample Size 
40 



Unit Cost 



Total Program 
Cost 



Development 

Investment 



Program Length 



Sample Size 
7 



SOLE-SOURCE CONTRACTED PROGRAMS 



Performance 

Achievement 


Cost 

Growth (%) 


Cost 

Growth (S) 


Schedule 
Slip (%) 


Schedule 
Slip (Mos . ) 


0.2001 


-0.3363 

(0.013) 


0.1433 


-0.1103 


0.0243 


0.2042 


-0.3304 

(0.020) 


0.5929 

(0.001) 


-0.4943 

(0.001) 


-0.3401 

(0.017) 


0.0392 


-0.1106 


-0.0598 


-0.0448 


-0.0539 


OO 

OO CO 

OO o 

CM • 

< o 
o ^ 


-0.1480 


0.2477 

(0.067) 


-0.2710 

(0.050) 


0.0464 




Kean Outcomes 






Perf 

0 


ornance 

.986 


Cost Growth 
1.207 




Schedule Slip 
1.303 




TA3LE 3 - 8(B) 








MULTIPLE 


-SOURCE CONTRACTED PROGRAMS 






Performance 

Achievement 


Cost 

Growth (%) 


Cost 

Growth (3) 


Schedule 
Slip (%) 


Schedule 
Slip (Mos. ) 


-0.2041 


-0.6429 

(0.060) 


-0.2357 


-0.5357 


-0.3571 


-0.6124 

(0.072) 


-0.4643 


0.0357 


-0.6429 

(0.060) 


-0.5714 

(0.090) 


0.6124 

(0.072) 


-0.2143 


0.3571 


-0.1786 


-0.0714 


0.0 


-0.4140 


-0.1261 


-0.6847 


-0.5225 



(0.045) 



Performance Cost Growth Schedule Slip 

1.023 1.140 1.443 
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TABLE 3 - 9(A) 



PROGRAMS IN PROCUREMENT 





Performance 


Cost 


Cost 


Schedule 


Schedule 




Achievement 


Growth (°’,) 


Growth ( $ ) 


Slip (%) 


Slip (Mos.) 


Unit Cost 


0.0901 


-0.4680 


-0.2348 


0.0462 


0.1611 






(0.008) 


(0.079) 






Total Program 


0.0619 


-0.3233 


0.4188 


-0.6198 


-0.5926 


Cost 




(0.054) 


(0.017) 


(0.001) 


(0.001) 


Development 


-0.1763 


-0.2361 


-0.1229 


-0.4241 


-0.0675 


Investment 








(0.014) 




Program Length 


0.3805 


-0.2419 


-0.0589 


-0.3564 


-0.1602 




(0.028) 






(0.037) 




Sample Size 


Perfi 


ormance 


Cost Growth 




Schedule Slip 


27 


0 


.976 


1.225 




1.333 







TABLE 3 


- 9(B) 








PROGRAMS IN DEVELOPMENT 








Performance 


Cost 


Cost 


Schedule 


Schedule 




Achievement 


Growth 


(%) Growth (5) 


Slip (%) 


Slip (Mos.) 


Unit Cost 


0.3953 


-0.1508 


0.3225 


0.0270 


0.0418 




(0.058) 










Total Program 


0.3293 


-0.4071 


0.4414 


-0.4880 


-0.0984 


Cost 


(0.098) 


(0.052) 


(0.033) 


(0.023) 




Development 


0.5151 


-0.3656 


0.0319 


-051501 


-0.0486 


Investment 


(0.017) 


(0.074) 








Program Length 


0.3304 


-0.3544 


0.3790 


-0.2931 


0.1953 




(0.098) 


(0.081) 


(0.067) 












Mean Outcomes 






Sample Size 


Performance 


Cost Growth 


Schedule Slip 


17 


0 . 


995 


1.178 




1.293 
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TABLE 3 - 10(A) 



NAVY PROGRAMS 





Performance 

Achievement 


Cost 

Growth (%) 


Cost 

Growth ($) 


Schedule 
Slip (%) 


Schedule 
Slip (Mos.) 


Unit Cost 


0.3760 

(0.032) 


-0.5484 

(0.003) 


-0.2130 


-0.1329 


-0.0185 


Total Program 
Cost 


0.2866 

(0.032) 


-0.6306 

(0.001) 


0.1304 


-0.6177 

(0.001) 


-0.5712 

(0.001) 


Development 

Investment 


0.0557 


0.0187 


-0.2331 


-0.1079 


-0.0922 


Program Length 


0.4317 

(0.016) 


-0.5273 

(0.004) 


-0.1628 
Mean Outcomes 


-0.4471 

(0.013) 


-0.2337 


Sample Size 
25 


Performance 

0.969 


Cost Grow 
1.272 


th ; 1 


Schedule Slip 
1.404 







TABLE 3 - 10(B) 










ARMY PROGRAMS 










Performance 

Achievement 


Cost 

Growth (%) 


Cost 

Growth (S) 


Schedule 
Slip (%) 


Schedule 
Slip (Mos.) 


Unit Cost 


-0.0305 


-0.2992 


0.0374 


-0.1923 


0.0 


Total Program 
Cost 


-u .1997 


-0.3036 


0.5604 

(0.019) 


-0.5110 

(0.037) 


-0.0718 


Development 

Investment 


0.0113 


-0.3297 


-0.3128 


0.0826 


0.2188 


Program Length 


0.2744 


-0.1506 


-0.1084 


-0.6177 


0.0501 



( 0 . 012 ) 

Mean Outcomes 

Sample Size Performance Cost Growth Schedule Slip 

14 1.025 1.149 1.292 
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TABLE 3 - 10 1C) 



Unit Cost 



Total Program 
Cost 



Development 

Investment 



Program Length 



Sample Size 
9 



USAF PROGRAMS 



Performance 

Achievement 


Cost 

Growth (%) 


Cost 

Growth ($) 


Schedule 
Slip (? 0 ) 


Schedule 
Slip (Mos . ) 


0.1219 


0.1157 


0.4167 


-0.0848 


0.0509 


0.3133 


0.0 


0.7333 

(0.012) 


-0.2034 


-0.0170 


-0.6659 

(0.027) 


0.1303 


0.2678 


0.1532 


0.2383 


0.2283 


-0.0952 


0.5283 


-0.0479 


0.4072 



(0.091) 

Mean Outcomes 

Performance Cost Growth Schedule Slip 

1.006 1.141 1.160 
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Table 3-6 has displayed the correlation results for all 
forty-eight programs without any differentiation. The 
strength of program cost in determining schedule success is 
apparent by the significance levels of the correlations 
(0.001 and 0.003). Table 3-7 reveals the superiority of 
the missile system correlation over the other system types. 
Eleven out of eighteen valid correlations were significant 
in the case of missile programs, while ship systems produced 
but five. Oddly enough, aircraft systems with a larger sample 
size than ship systems had only four significant correlations. 
Apparently, aircraft systems were responding to other deter- 
inants, most probably managerial in origin. 

Despite the great difference in sample size note pre- 
viously, multi-source programs and sole-source programs pro- 
duced very nearly the same number of significant correlations. 
(See Table 3-8). In the case of the second source programs, 
the lack of impact on the part of program cost on absolute 
cost growth rioted earlier was suprising, thereby suggesting 
that cost benefits do occur as the rssult of competition. In 
common with multi-source contracted programs, developing 
programs reflected a decreasing level of performance with 
increasing technical risk. This expected result was 
curiously not demonstrated elsewhere. The differences 
between service programs discussed earlier are exhibited in 
Table 3-10. Not suprisingly, with a larger sample size, 

Navy programs produced more significant correlations than 
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either of the other services' programs combined. 

Table 3-11 represents a summary of the statistically 
significant ( 0.100) tests presented in Tables 3-7 through 

3 - 10, showing the degree to which variable relationships 
were common regardless of the controlled program character- 
istic. Similarily, Table 3-12 displays a summary of the 
correlation (not shown) of dependent variables with them- 
selves and the degree of commonality encountered. 
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TABLE 3-11 



SUMMARY OF SIGNIFICANT CORRELATIONS BETWEEN 
DEPENDENT/INDEPENDENT VARIABLES 



Performance 


Positive Negative 


Unit Cost 
Program Cost 
Development Investment 
Program Length 


1 2 

9 0 

0 . 0 
4 0 


Cost Growth (%) 


Positive Negative 


Unit Cost 
Program Cost 
Development Investment 
Program Length 


0 6 

0 7 

0 2 

0 3 


Schedule Slip (%) 


Positive Negative 


Unit Cost 
Program Cost 
Development Investment 
Program Length 


2 0 

3 1 

2 0 

6 0 


Schedule Slip (Mos . ) 


Positive Negative 


Unit Cost 
Program Cost 
Development Investment 
Program Length 


0 1 

0 10 

1 1 

0 7 


Cost Growth ($) 


Positive Negative 


Unit Cost 
Program Cost 
Development Investment 
Program Length 


1 1 

0 7 

2 1 

0 1 
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TABLE 3-12 



SUMMARY OF SIGNIFICANT CORRELATIONS BETWEEN 
DEPENDENT VARIABLES 



Performance 



Positive 



Negative 



Cost Growth (%) 0 
Cost Growth ($) 0 
Schedule Slip (%) 0 
Schedule Slip (Mos.) 0 



6 

0 

0 

0 



Cost Growth (%) 



Positive 



Negative 



Cost Growth (?) 10 
Schedule Slip (%) 9 
Schedule Slip (Mos.) 4 



0 

0 

0 



Cost Growth ($) 



Positive 



Negative 



Schedule Slip (%) 1 

Schedule Slip (Mos.) 1 



2 

2 



Schedule Slip (%) 



Positive 



Negative 



Schedule Slip (Mos.) 



12 



0 



The results of the correlation analysis as summarized in 
the above tables illustrate some interesting points. The 
first of these is the weakness of the independent variable, 
development investment. Construed as a measure of tech- 
nological risk, the variable failed to correlate with any of 
the outcomes except in a few cases. One of two conclusions 
can be reached, either the program character sitic of risk 
is valid and the variable as defined did not capture it, or 
risk has little or no effect on program outcome. It is 
suspected that the former is true based upon the results 
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reported by Peck and Scherer. 17 Also, on reflection, it is 
felt that the data as presented in the SAR*s could not be 
manipulated otherwise to measure risk. 

Much the same criticism can be directed toward the use 
of program unit cost. With the exception of relative cost 
growth, unit cost did little to explain program outcomes. 

The fact that unit cost did correlate significantly with 
relative cost growth in six out of a possible twelve cases 
was unexpected. The variable was intended initially to 
support development investment as a measure of the complex- 
ity of the program. Intuitively one would expect that the 
relationship between unit cost and cost growth (%) to be a 
positive one, not negative. It is suspected that unit cost 
may be reflective of an unknown project feature. 

From Table '3 - 12, a summary of the correlations between 
dependent variables reveals that a definite relationship 
between cost and schedule growth exists, perhaps more aptly 
put as "time is money". As in the case of the analysis 
involving both dependent and independent variables, perfor- 
mance achievement remained a largely unexplained outcome, 
although in 6 of 11 trials, a significant negative correla- 
tion between relative cost growth and performance was report- 
ed. It may be hasty to say that a desire to reach or surpass 
initial performance levels causes cost overruns since a 

17 Peck, M. J. and Scherer, F. M., The Weapons Acquisi- 
tion Process: An Economic Analysis, Chapter 16, Research 
Division, Graduate School of Business Administration, Harvard 
University, 1962 
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corresponding relationship between performance and cost 
growth in dollars was not evident. The overwhelming 
number of significant correlations between relative and 
absolute cost overruns, and schedule slippages must be 
discounted on definitional grounds. 

D. CONCLUSIONS 

From the results of the hypothesis testing, three 
major conclusions have been derived. 

1 . Contrary to previous belief, large, expensive 
programs exhibited a greater controllability in terms 
of schedule than did projects of smaller size . 

One possible explanation is that large, expensive pro- 
grams attract a great amount of attention from every quarter, 
the sponsoring service itself, OSD, Congress and the public. 
Conceivably, greater pressure are brought to bear to keep 
large programs within the limits of an original assessment 
as suggested .by the Blue Ribbon Defense Panel, and thereby 
avoid the embarassment of adverse program outcomes. A 
second reason (and one not necessarily exclusive of the 
first) might be that large programs, because of their size 
and sensitivity, are subject to greater scrutiny before the 
projects get underway, and thus the original estimates are 
more realistic. Also, once underway, program offices may 
find themselves more adequately staffed both in numbers of 
people and their expertise to ensure success. 
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2 . Despite indications that early and substantial 
funding may reduce the scope of future cost growth , 
there is little clearcut effect of "front-end" money on 
the technical and operational outcomes of the program . 
Although no judgements were made of the manner in which 
development funds were spent, there are indications that 
the management of the development/system definition phase 
of the life-cycle should be redirected. The staff report on 
the acquisition process to the Blue Ribbon Defense Panel in 
1970 reported that the services had failed to receive 
sufficient funding to carry out necessary technical work 
during the early periods of their programs. What would often 
happen then was that critical engineering studies would be 
conducted during the production phase when money was more 
abundant. It is plausible that an adverse effect on system 
capabilities and performance would be suffered. 

Related to this last point is the complete lack of effect 
that development investment displayed when defined as a 
ratio of development funding to program cost. It may be 
entirely possible that the definition of development invest- 
ment is inadequate to the task of measuring technical 
uncertainty. Despite the fact that a majority of programs 
(40 out of 48) met or surpassed the initial performance goals 
established for them, and that the “science" of predicting 
what is technically feasible may be more efficient than the 
"art" of estimating cost or schedule outcomes, one should 
be reluctant to surmise that developmental effort has less 
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effect on project success than other program variables. 
Although certainly less program resources are committed 
to advancing the state-of-the-art than manufacturing final 
hardware, judging from figures for the cost growth of 
strictly development budgets as reported by Peck and Scherer, 
cost variance has exceeded three or four times the original 
estimates in the past. It is conceivable that these develop- 
mental excesses could have a snowballing effect of the 
remainder of the program, expecially on schedule milestones 
if stirct measures against concurrent development and pro- 
duction are taken. 

3 . Program length is related inversely to the achieve- 
ment goals . 

It appears from an examination of all the characteristics 
that performance estimates were either met or exceeded, and 
so it is surprising to find indications that program length 
is related in a positive way to achievement of desired 
performance. Peck and Scherer came across a possible explan- 
ation in their study. 

Each engineer wants to perfect things on which he 
works. Every development nevertheless involves 
compromises between quality- and time. If schedules 
are set too generously, there is a tendency for 
persons working on the progect to use up the sche- 
duled time and still be no nearer completion on 
their hardware goals than under a shorter schedule. 
Therefore, in order to provide a good target a 
schedule should be short enough to make people think 
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and worry about getting the hardware out. 
Unfortunately, program length contributed very little else to 
an understanding of program success. 



18 Peck, M. J. and Scherer, F. M., The Weapons Acquisition 
Process; An Economic Process , p. 413, Research Division, 
Graduate School of Business Administration, Harvard University, 
1962 
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IV. COMMENTS AND RECOMMENDATIONS 



Fundamental to this thesis was the point of view from 
which the analysis was conducted. Each of the forty-eight 
programs were considered as "black boxes" into which certain 
inputs were made (cost, time, effort, etc.) and from which 
certain measurable outputs were collected. Meadows and 
Marquis criticized such "post mortem" techniques in their 
own analysis of the implications of forecasting errors for 
ignoring the interaction between forecasts and outcomes in 
the process of development. 19 Additionally, Determinants 
of Project Success by Murphy, et al, stated, "with rare 
exception, the determinants of project success were manage - 
ment factors, things which management had the potential 
ability to inf luence . . . " , 20 a conclusion which necessarily 
required a point of view from within the box. 

The analysis undertaken here does not mean to suggest 
that management techniques are without effect in determining 
program outcomes. Instead, the study asked how program 
initial conditions related to program outcomes, or more 
precisely, to what extent could the initial conditions be 

19 Meadows, D. L. and Marquis, D. G., Characteristics 
and Implications of Forecasting Errors in the Selection of 
R&D Projects , paper delivered by D . L. Meadows at The 
Second Annual Technology and Management Conference, Washing- 
ton, D.C., 21 March 1968 

20 Murphy, D. C., Baker, B. N., and Fisher, D., Deter- 
minants of Project Success , p. 5, Management Institute, 
School of Management, Boston College, 1974 
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relied upon to "explain" program success. 'Explain' is 
used here in the sense of signalling the degree of success 
that a program most likely would enjoy. And because of 
that. Meadows and Marquis are correct, the transfer function 
of the black box is not really of primary concern. Actually, 
the results of this study do uphold Murphy in that the 
weakness of the correlation implies that other determinants 
are at work. 

Before dismissing the methodology as indecisive at best, 
perhaps a few suggestions as to how to improve it would be 
in order. During the accumulation of data for the thesis, 
a concerted effort was made to ascertain a measure of the 
relative importance of each project without success. The 
Navy practice of assigning importance classification based 
on the significance of the preceived threat to program once 
the Specific Operational Requirement has been drafted is no 
longer in effect, and no joint ranking system appears to 
exist. The addition of this extra variable as a measure of 
how the program is perceived outside the project office 
would strengthen greatly the correlation prodedure and no 
doubt serve as the basis for an interesting hypothesis as 
well . 

It is felt that the data from the SAR should be supple- 
mented by information gained from other sources, such as 
interviews or questionaires . For instance, the dilemma as 
to what effect technological advance has on project success 
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could be settled by correlating the technical assessment of 
program office engineering personnel with the SAR measure- 
ment. Additionally, some understanding of how initial 
forecasts drive the program, that is how the attitude of 
managers toward these development estimates is reflected in 
the outcome, might prove an interesting variable in itself. 
Obviously though, the distance between the analyst and the 
programs would have to be shortened . 
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The start of the MPE was delayed two months due to design, flight testing and production 
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COST VARIANCE ANALYSIS 
(Dollars in Millions) 
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APPENDIX B 



COMPUTATION OF IMPLICIT PRICE DEFLATORS 

The Direct Budget Plan (TOA) History (following page) 
was the source of the implicit price deflators discussed 
earlier. The following example will demonstrate how these 
conversion factors were derived. 

A - 7E Program 

From the SAR in Appendix A, the program base 
year is established as 1967. Then, from the TOA history, 
the ratio of FY 1967 obligations to an equivalent amount of 
1977 dollars is computed. This is: 

$21,19 OM/ $ 4 2 , 9 0 4M = 0,493 

Similarily, a ratio of FY 1974 obligations to 1977 dollars 
is also calculated. This is: 

$26 , 88 8M/$32 , 926M = 0,817 

The second ratio is then divided by the first and the result 
is a price deflator for converting 1967 dollar amounts to an 
equivalency in 1974 dollars. 

0.817/0.493 = 1,66 
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Department of the Navy 

DIRECT BUDGET PI, AN (TOA) HISTORY 
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APPENDIX C 



COMPUTATION OF OUTCOME VARIABLES 



Listed in the following pages are the forty-eight programs 
which constituted the sample of acquisition programs studied 
and their associated outcomes. Before proceeding further, 
it was felt that some examples of how each of the outcome 
factors was obtained would be beneficial. The A-7E SAR in 
Appendix A will serve as a data source, 

1. Cost Growth (%) 

As stated previously in Chapter II, the measure of 
relative cost growth was obtained directly from "SAR Program 
Acquisition Cost Summary" . That document reported the A-7E 
project relative cost growth as + 26%. Thus, the program has 
a cost growth- factor of 1.26. 

2. Performance 

In Chapter II, the achieved performance factor is 
stated as being derived as a weighted average of key per- 
formance criteria. In the case of the A-7E program, these 
were : 

1. maximum speed at sea level; 

2. radius/range during primary attack mission; 

3. ceiling/altitude; 

4. reliability; 

5. maintainability - 

a. Scheduled Depot Level Maintenance (SDLM) 

b. Maintenance Man-Hour per Flight Hour. 
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A comparison of the current estimate/demonstrated performance 
and the development estimate for each of the criteria above 
yields the following results. 

1. 505 KTS/555 KTS =0.91 

2. 537 NM/ 475 NM = 1.13 

3. 35,500 ft/35,000 ft = 1.00 

4. 0.78/0.84 - 0.93 

5a. 24 MOS/30 MOS =0.80 
b. 5.4 HR/9.6 HR = 0.56 

(Weighted equally, maintainability 
is 0.68) 

Summing all of the scores and dividing by the number of 
criteria, one achieves a performance factor of 0.93 for the 
program. 

The above procedure is illustrative of the general 
approach used. There were two modifications to this method, 
one in the case of ships, the other in the case of certain 
one-shot devices. Unlike aircraft where speed, maneuver- 
ability, range, reliability and maintainability taken toge- 
ther represent a meaningful measure of system performance, 
a comparison of SAR criteria for ships (e.g., speed, manning 
level, displacement, etc.) does not. Invariably these 
criteria are met. The existence of multiple mission areas for 
a single ship type further complicates the figure of merit 
calculations. This is a good example of where system 
ewectiveness breaks down in a computational sense. For these 
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reasons, ship systems were arbitrarily assigned a performance 
factor of 1.0. 

One-shot systems (less strategic systems) were evaluated 
based on a comparison of their single shot probability of 
kill given in the SAR. This assumed the systems were activated 
within their proper search/attack envelopes and that all 
components functioned properly. 

3. Schedule Growth (%) 

From the Schedule Milestone section of the SAR, the 
program was initiated in May 1963 and achieved an IOC on 
schedule in April 1970. Thus, there was no schedule slippage 
and a factor of 1.0 was awarded. 
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PROGRAM 

NAVY 

Program 

1. S-3A 

2. Trident 

3. Poseidon C-3 

4. SSN 688 

5. Trident C-4 

6. DD 963 

7. CGN 38 

8 . Harpoon 

9. LHA 

10. CVAN 68 

11. CVAN 69 

12. Condor 

13. CH-53E 

14. MK 48-1 . . 

15. P-3C 

16. Captor 

17. Sparrow III 

18. PHM 

19. Phoenix 

20. FFG 7 

21. A-7E 

22. E-2C 



OUTCOME SUMMARY BY SERVICE 



Performance 



1.03 

0.99 

N/A 

1.01 

1.06 

1.05 

1.05 

1.10 

1.18 

1.16 

1.25 
1.46 
1.08 
1.19 
1.17 
1.56 
1.64 
1.84 
1.43 
1,45 

1.26 
1.36 



Schedule Slip(%) 

1.00 

1.14 
1,07 

1.15 
1.13 
1.24 
1.24 

1.13 
1.87 
1.35 
1.24 
3.29 

1.32 

1.14 

1.05 

1.33 
2.80 
1.41 

1.06 
1.13 
1.00 
1.03 



Cost Growth (%) 



1.04 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.94 

0.97 

0.98 

0. 99 

1 . 00 
1.00 
1.00 
1.00 
1.00 
0.93 
0.89 
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Program 


Performance 


Cost Growth (%) 


Schedule Slip(%) 


23. 


Phalanx 


0.85 


1.37 


1.36 


24. 


F-15A 


0.84 


1.22 


1.16 


25. 


Sidesinder 
(AIM 9-L) 


0.79 


1.61 


2.45 


MEAN 


0.969 


1.272 


1.404 


ARMY 










Program 


Performance 


Cost Growth (%) 


Schedule Slip(%) 


1 . 


XM 198 


1.16 


1.03 


1.30 


2. 


Stinger 


1.62 


1.22 


1.44 


3. 


Hawk 


1.04 


1.39 


1.25 


4. 


UTTAS 


1.00 


0.88 


1.02 


5. 


Roland 


1.00 


1.00 


1.23 


6 . 


XM 1 


1.00 


1.08 


1.00 


7. 


Patriot ' 


1.00 


1.07 


1.21 


8. 


AAH 


1.00 


1.15 


1.33 


9. 


MICV 


1.00 


1.19 


1.34 


10. 


Dragon 


1.00 


1,19 


1.50 


11. 


Copperhead 


0.93 


1.18 


1.11 


12. 


TOW 


0.86 


1.21 


1.41 


13. 


Lance 


0.86 


1.12 


1,65 


14. 


TACFIRE 


0.88 


1.38 


N/A 


MEAN 


1.025 


1.149 


1.292 
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USAF 



Schedule Slip(%) 

1.00 



Program 


Performance 


Cost Growth (%) 


1. F-16 


1.06 


1.00 


2. B-l 


1.12 


1.13 


3. A- 10 


1.03 


1.22 


4. E-3A 


1.02 


1.16 


5. EF-llA 


1.00 


0.94 


6. Maverick 


1.00 


1.02 


7. E-4 


1.00 


1.40 


00 

• 

i 

H 


1.00 


1.25 


9. Minuteman III 


0.82 


1.15 


MEAN 


1.006 


1.141 


OVERALL MEAN 


0.992 


1.210 



1.35 

1.13 

1.10 

1.00 

1.22 

1.61 

1.03 

1.00 

1.160 

1.326 
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SAR PROGRAM ACQUISITION COST SUMMARY 
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